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CLAIMS 



What is claimed is: 

L A computer implemented method for determining the semantic meaning of images, 
comprising: 

deriving a set of perceptual semantic categories for representing important semantic cues 
in the human perception of images, where each semantic category is modeled through a 
combination of perceptual features that define the semantics of that category and that 
discriminate that category from other categories; and 

for each semantic category, forming a set of the perceptual features as a complete feature 
set CFS. 

2. A method as in claim 1, wherein the perceptual features and their combinations are derived 
through subjective experiments performed with human observers. 

3. A method as in claim 1 , further comprising extracting perceptual features from an input image 
and applying a perceptually-based metric to determine the semantic category for that image. 

4. A method as in claim 3, comprising processing the input image to compute the CFS; comparing 
the input image to each semantic category through the perceptually-based metric that computes 
a similarity measure between the features used to describe the semantic category and the 
corresponding features extracted from the input image; and assigning the input image to the 
semantic category that corresponds to a highest value of the similarity measure. 

5. A method as in claim 1, further comprising computing features from the CFS for images in an 
image database; and generating a distance measure for characterizing a relationship of a selected 
image to another image in the image database by applying a perceptually-based similarity metric. 

6. A method as in claim 5, where values of the similarity metric computed for images in the 
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image database are subsequently used to search for similar images in the image database. 

7. A method as in claim 5, where values of the similarity metric computed for images in the 
image database are subsequently used to organize images in the image database. 

8. A method as in claim 5, where values of the similarity metric computed for images in the 
image database are subsequently used to display images in the image database in an organized 
manner. 

9. A method as in claim 5, and further comprising defining a subset of features for the selected 
image or for an image retrieved from the image database, and using the subset of features to 
refine a search through the image database. 

10. A method as in claim 5, wherein the image database is located at a remote location and is 
reachable through a data communications network. 

11. A method as in claim 5, wherein the image database is located at a remote location and is 
reachable through a data communications network, and where the step of characterizing the 
relationship of the selected image to another image in the image database by applying the 
perceptually-based similarity metric is accomplished to retrieve an image from the remote image 
database. 

12. A method as in claim 5, wherein the image database is located at a remote location and is 
reachable through a data communications network, and where the step of characterizing the 
relationship of the selected image to another image in the image database by applying the 
perceptually-based similarity metric is accomplished in conjunction with a text-based search 
algorithm to retrieve a multi-media object from the remote location. 

13. A method as in claim 3, wherein to assign a particular semantic category to an image all of 
a set of Required Features must be present in the image, and at least one of a set of Frequently 
Occurring Features must be present in the image. 
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14. A data processing system comprising a data processor, a graphical user interface and a 
memory that stores a collection of digital images in an image database, said data processor 
operating in accordance with a stored program for determining the semantic meaning of images 
in accordance with a set of perceptual semantic categories that were previously derived from 
human observers and that represent important semantic cues in the human perception of images, 
where each semantic category is modeled through a combination of perceptual features that define 
the semantics of that category and that discriminate that category from other categories, where 
for each semantic category there exists a set of the perceptual features as a complete feature set 
CFS, said data processor extracting perceptual features from an input image and applying a 
perceptually-based metric to determine the semantic category for the input image, 

1 5 . A system as in claim 14, where said data processor processes the input image to compute the 
CFS; compares the input image to each semantic category through the perceptually-based metric 
that computes a similarity measure between the features used to describe the semantic category 
and the corresponding features extracted from the input image and assigns the input image to the 
semantic category that corresponds to a highest value of the similarity measure. 

16. A system as in claim 14, where said data processor computes features from the CFS for 
images in an image database; and generates a distance measure for characterizing a relationship 
of a selected image to another image in the image database by applying the perceptually-based 
similarity metric. 

17. A system as in claim 16, where values of the similarity metric computed for images in the 
image database are subsequently used to search for similar images in the image database. 

18. A system as in claim 16, where values of the similarity metric computed for images in the 
image database are subsequently used to organize images in the image database. 

19. A system as in claim 16, where values of the similarity metric computed for images in the 
image database are subsequently used to display images from the image database in an organized 
manner. 
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20. A system as in claim 16, where said data processor cooperates with said graphical user 
interface for enabling a user to define a subset of features for the selected image or for an image 
retrieved from the image database, and subsequently uses the subset of features to refine a search 
through the image database. 

21. A system as in claim 16, wherein the image database is located at a remote location and is 
reachable through a data communications network that is bidirectionally coupled to said data 
processor through a network interface. 

22. A system as in claim 21, where the data processor applies the perceptually-based similarity 
metric to retrieve an image from the remote image database. 

23. A system as in claim 21, where the data processor applies the perceptually-based similarity 
metric to in conjunction with a text-based search algorithm to retrieve a multi-media object from 
the remote location, 

24. A system as in claim 14, wherein for said data processor to assign a particular semantic 
category to an image all of a set of Required Features must be present in the image, and at least 
one of a set of Frequently Occurring Features must be present in the image. 

25. A computer program embodied on a computer readable media for directing a computer to 
execute a method for processing digitally represented images, comprising program instructions 
for processing a set of perceptual semantic categories for representing semantic cues related to 
the manner in which human observers perceive and organize images, the semantic categories 
being modeled using multidimensional scaling and hierarchical clustering techniques and 
comprising a combination of perceptual features that define the semantics of a particular category 
and that discriminate that category from other categories, where the perceptual features and their 
combinations are derived through subjective experiments performed with human observers; for 
each semantic category, program instructions for forming a set of the perceptual features as a 
complete feature set CFS and, responsive to an input image, program instructions for determining 
a CFS of the input image and for using the determined CFS to compare the input image to images 
stored in an image database. 



33 

26. A computer program as in claim 25, where as a result of comparing the input image to images 
stored in the image database one or more most similar images are identified in the image 
database. 

27. A computer program as in claim 25, where as a result of comparing the input image to images 
stored in the image database one or more most similar images from the image database are 
displayed. 

28. A computer program as in claim 25, wherein the step of using the determined CFS includes 
using a similarity metric to assign a semantic category to the input image, where the similarity 
metric operates such that all of a subset of Required Features of the semantic category must be 
present in the input image, and at least one of a subset of Frequently Occurring features of the 
semantic category must be present in the input image. 

29. A computer implemented method for processing digitally represented images, comprising: 

obtaining a set of perceptual semantic categories for representing semantic cues related 
to the manner in which human observers perceive and organize images, the semantic 
categories being modeled using multidimensional scaling and hierarchical clustering 
techniques and comprising a combination of perceptual features that define the semantics 
of a particular category and that discriminate that category from other categories, where 
the perceptual features and their combinations are derived through subj ective experiments 
performed with human observers; 

for each semantic category, forming a set of the perceptual features as a complete feature 
set CFS; and 

for individual ones of images stored in an image database, determining a CFS of each 
image and classifying each image by using a similarity metric for assigning a semantic 
category to the image, where the similarity metric operates such that all of a subset of 
Required Features of the semantic category must be present in the image, and at least one 
of a subset of Frequently Occurring features of the semantic category must be present in 
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the image. 

30. A method as in claim 29, further comprising identifying a query image; determining a CFS 
of the query image; and using the determined CFS to compare the query image to the images 
stored in the image database. 

3 1 . A method as in claim 30 where as a result of comparing, one or more most similar images to 
the query image are identified in the image database. 

32. A method as in claim 30 where as a result of comparing, one or more images from the image 
database are displayed. 

33. A method as in claim 29, and further comprising displaying images from the image database 
organized by semantic category. 

34. A method as in claim 29, and further comprising operating a user interface to browse through 
the images from the image database by using the semantic categories. 

35. A method as in claim 30 where as a result of comparing at least one image is returned from 
the image database, and further comprising processing the returned image to select a portion of 
the returned image, computing a CFS of the selected portion of the returned image, and using the 
computed CFS to locate at least one further image in the image database. 

36. A method as in claim 29, further comprising identifying a query image; determining a CFS 
of the query image; and using the determined CFS to compare the query image to the images 
stored in the image database, where the image database is remotely stored and is reachable 
through a data communications network. 

37. A method as in claim 29, further comprising identifying a query image; determining a CFS 
of the query image; and using the determined CFS to compare the query image to the images 
stored in the image database, where the image database is accessed via a server coupled to the 
internet. 



35 

38. A method as in claim 37, where the query image forms a part of a query that also includes a 
textual component. 
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